Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

community[minor]: Refactoring PyMuPDF parser, loader and add image blob parsers #29063

Merged
merged 47 commits into from
Jan 20, 2025
Merged

community[minor]: Refactoring PyMuPDF parser, loader and add image blob parsers #29063

merged 47 commits into from
Jan 20, 2025
+2,385 −186

Conversation

pprados
Copy link
Contributor

@pprados pprados commented Jan 7, 2025 •
edited by eyurtsev
Loading

  • Adds BlobParsers for images. These implementations can take an image and produce one or more documents per image. This interface can be used for exposing OCR capabilities.
  • Update PyMuPDFParser and Loader to standardize metadata, handle images, improve table extraction etc.
  • Twitter handle: pprados

This is one part of a larger Pull Request (PR) that is too large to be submitted all at once.
This specific part focuses to prepare the update of all parsers.

For more details, see PR 28970.

@vercel Vercel
Copy link

vercel bot commented Jan 7, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
langchain ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jan 20, 2025 4:17pm

@pprados pprados marked this pull request as ready for review January 7, 2025 09:16
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. community Related to langchain-community Ɑ: doc loader Related to document loader module (not documentation) labels Jan 7, 2025
@pprados pprados force-pushed the pprados/02-pymupdf branch from 039819c to 3beda82 Compare January 7, 2025 16:09
@pprados
Copy link
Contributor Author

pprados commented Jan 7, 2025

@eyurtsev I rebase the code with master ;-)

eyurtsev
eyurtsev reviewed Jan 8, 2025
View reviewed changes
Copy link
Collaborator

@eyurtsev eyurtsev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great will take a look in the AM

@pprados pprados mentioned this pull request Jan 8, 2025
Draft
2 tasks
eyurtsev
eyurtsev reviewed Jan 9, 2025
View reviewed changes
Copy link
Collaborator

@eyurtsev eyurtsev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left two major comment, a few stylistic comments and some nits.

Let's tackle the two major comments:

  1. Define the standardized structure of metadata
  2. Create a dedicated ImageParser which is a blob parser

libs/community/langchain_community/document_loaders/parsers/pdf.py Outdated Show resolved Hide resolved
libs/community/langchain_community/document_loaders/parsers/pdf.py Show resolved Hide resolved
libs/community/langchain_community/document_loaders/parsers/pdf.py Show resolved Hide resolved
libs/community/langchain_community/document_loaders/parsers/pdf.py Outdated Show resolved Hide resolved
libs/community/langchain_community/document_loaders/parsers/pdf.py Show resolved Hide resolved
libs/community/langchain_community/document_loaders/parsers/pdf.py Outdated Show resolved Hide resolved
libs/community/langchain_community/document_loaders/parsers/pdf.py Outdated Show resolved Hide resolved
libs/community/langchain_community/document_loaders/parsers/pdf.py Show resolved Hide resolved
libs/community/langchain_community/document_loaders/parsers/pdf.py Outdated Show resolved Hide resolved
libs/community/langchain_community/document_loaders/parsers/pdf.py Outdated Show resolved Hide resolved
@pprados pprados marked this pull request as draft January 10, 2025 12:45
@pprados pprados force-pushed the pprados/02-pymupdf branch from 0d99673 to 3fe4ec5 Compare January 10, 2025 13:40
@pprados pprados force-pushed the pprados/02-pymupdf branch 2 times, most recently from 4342991 to 760267b Compare January 10, 2025 14:05
@pprados
Copy link
Contributor Author

pprados commented Jan 17, 2025

yum is deprecated and replaced by dnf.
But, il doc/Makefile, yum is used.
I can not install yum on Ubuntu.
It's difficult for me to fix a bug in the documentation

@pprados pprados force-pushed the pprados/02-pymupdf branch from a9065ee to 0fe40a2 Compare January 17, 2025 13:32
@pprados pprados force-pushed the pprados/02-pymupdf branch from 0fe40a2 to 2c4b1f6 Compare January 17, 2025 13:49
@pprados pprados force-pushed the pprados/02-pymupdf branch from 2c4b1f6 to a4587f0 Compare January 17, 2025 13:52
@pprados pprados force-pushed the pprados/02-pymupdf branch from f2fea1f to 74d3617 Compare January 18, 2025 07:25
@dosubot dosubot bot added the lgtm PR looks good. Use to confirm that a PR is ready for merging. label Jan 20, 2025
@eyurtsev eyurtsev changed the title Refactoring PDF loaders: 02 PyMuPDF community[minor]: Refactoring PDF loaders: 02 PyMuPDF Jan 20, 2025
@eyurtsev eyurtsev changed the title community[minor]: Refactoring PDF loaders: 02 PyMuPDF community[minor]: Refactoring PyMuPDF parser, loader and add image blob parsers Jan 20, 2025
@eyurtsev eyurtsev merged commit 4efc509 into langchain-ai:master Jan 20, 2025
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eyurtsev eyurtsev eyurtsev approved these changes

Assignees

@eyurtsev eyurtsev

Labels
community Related to langchain-community Ɑ: doc loader Related to document loader module (not documentation) 🤖:docs Changes to documentation and examples, like .md, .rst, .ipynb files. Changes to the docs/ folder lgtm PR looks good. Use to confirm that a PR is ready for merging. size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
PR Reviews
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pprados @eyurtsev